Summary

Adds TreeSitterParserBase, a shared abstract base class that sits between AbstractLanguageParser and any future tree-sitter-backed language parser. This PR introduces the infrastructure layer only, no language-specific logic is included. The first concrete subclass (PythonParser) follows in a separate PR.

Objective

Vanir's parser pipeline expects every language parser to produce a ParseResults object containing:

• function_chunks - one FunctionChunkBase per function, carrying its name, parameters, return types, used data types, local variables, called functions, and raw tokens
• line_chunk - a LineChunkBase with all file tokens grouped by line number, used for line-based signature matching
• parse_errors - a list of ParseError for any syntax errors encountered

Existing parsers (C/C++, Java) fulfill this contract via ANTLR4 grammars. This PR fulfills the same contract using tree-sitter, enabling faster parser development for new languages with no grammar compilation step.

Implementation

tree_sitter_base.py

TreeSitterParserBase (class)

The core class. Subclasses set three class-level attributes:

• LANGUAGE - a tree_sitter.Language instance for the target language
• _FUNC_QUERY - a compiled tree-sitter Query for matching function definitions
• SKIP_TOKEN_TYPES - a frozenset of node types to exclude from token collection (e.g. comments)
• STRING_NODE_TYPE - the node type to treat as an opaque string token (default: 'string')

__init__(filename)

• Reads the source file as bytes
• Instantiates a tree-sitter Parser with the subclass's LANGUAGE
• Parses the file into a concrete syntax tree (self._tree)
• Tree-sitter elements used: Parser, tree_sitter.Language

get_chunks(affected_line_ranges_for_functions), main pipeline

Implements the full ParseResults assembly in four steps:

Step 1, Function discovery

• Runs _FUNC_QUERY against the root node using QueryCursor
• Extracts func.def and func.name capture groups from each match
• Tree-sitter elements used: QueryCursor, Query.matches(), named captures

Step 2, FunctionChunkBase construction

• For each matched function node, computes start/end line numbers from start_point / end_point
• Filters by affected_line_ranges_for_functions using _overlaps()
• Extracts function name from the func.name capture node
• Retrieves the parameters child field via child_by_field_name('parameters')
• Delegates to three abstract methods (see below) for language-specific extraction
• Identifies nested function byte ranges to exclude from local/call collection
• Collects flat token list via _flat_tokens_cursor()

Step 3, LineChunkBase construction

• Runs _collect_tokens_cursor() over the entire root node
• Produces a dict[int, list[str]] mapping line numbers to token lists
• Passed directly into LineChunkBase

Step 4, Parse error collection

• Runs _collect_errors_cursor() to find all ERROR nodes in the tree
• Each ERROR node becomes a ParseError with line, column, and token preview

Cursor utility functions

_collect_tokens_cursor(node, skip_types, string_type, out_dict)

• Iterative DFS using tree-sitter's TreeCursor (avoids Python recursion limits)
• Prunes subtrees whose root type is in skip_types (e.g. comments, decorators)
• Emits nodes of string_type as a single opaque token
• Emits all other leaf nodes (no children) as individual tokens
• Populates out_dict[line_number] → [token, ...]
• Tree-sitter elements used: node.walk(), TreeCursor.goto_first_child(), goto_next_sibling(), goto_parent(), node.text, node.start_point

_flat_tokens_cursor(node, skip_types, string_type)

• Thin wrapper over _collect_tokens_cursor()
• Returns a flat ordered list of token strings (sorted by line, then order within line)

_collect_errors_cursor(root)

• Iterative DFS that visits every node without pruning
• Collects all ERROR nodes and returns them as ParseError objects
• Truncates bad token display to 80 characters
• Tree-sitter elements used: node.type == 'ERROR', node.start_point, node.text

_overlaps(func_start, func_end, ranges)

• Pure utility, returns True if a function's line range overlaps any of the provided affected line ranges
• Empty ranges means no filter (include all functions)

Abstract methods (subclasses must implement)

_extract_param_names(params_node) → list[str]

Extracts ordered parameter name strings from a tree-sitter parameters node. Subclasses traverse the parameter node's children according to the language's grammar (e.g. identifier, typed_parameter, default_parameter).

_extract_annotations(func_node) → (return_types, used_data_types)

Extracts type annotation information from a function definition node:

• return_types - list of token lists from the return type annotation (empty if none)
• used_data_types - list of token lists, one per annotated parameter

_collect_locals_calls(body_node, nested_ranges) → (local_vars, called_fns)

Walks the function body node to collect:

• local_vars - set of locally assigned variable names
• called_fns - set of called function names
• nested_ranges - byte ranges of nested functions to exclude from collection, preventing inner-scope variables from leaking into the outer function's signature

Design decisions

• No module-level tree-sitter import, the base class is importable on Python 3.9 (where tree-sitter is not installed) without error. All tree-sitter imports are lazy, inside methods.
• Iterative cursor DFS, avoids Python's default recursion limit on deeply nested ASTs. All three traversal utilities use TreeCursor instead of recursive calls.
• Language-agnostic, zero Python-specific logic in this file. All language grammar knowledge lives in concrete subclasses.

Dependencies

This PR is independent and can be merged alongside:

• feat/python-parser-deps
• feat/python-parser-build
• bug/fix-subclass-discovery

It must be merged before feat/python-parser.